Blacks.
While the phrase has long been used as an interpersonal generalization, it has become known as a racial epithet targeted to blacks only recently. The earliest documentation of this meaning that I can find is from 1992, when presidential candidate
Ross Perot was
chided for telling an
NAACP audience about financial hardships befalling "your people."